Information processing apparatus and non-transitory computer readable medium storing information processing program

ABSTRACT

An information processing apparatus includes a receiving unit that receives an input picture obtained by capturing a landscape including an object, and an estimation unit that estimates an image represented by the landscape appearing in the input picture based on a learning model in which the input picture received by the receiving unit is input.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2018-224214 filed Nov. 29, 2018.

BACKGROUND (i) Technical Field

The present invention relates to an information processing apparatus anda non-transitory computer readable medium storing an informationprocessing program.

(ii) Related Art

JP2009-251850A discloses a product recommendation system including: apicture file in which a plurality of product pictures including one ormore products are stored; a product area extraction unit that extractsone product area by deleting a background area from each product pictureincluding one or more products; a picture information database thatstores a product ID and product information of each productcorresponding to the product area extracted from each product picture inassociation with the product picture; a picture feature acquisition unitthat acquires a picture feature of the product corresponding to theproduct area by calculating the picture feature of the extracted productarea; a feature database that stores the picture feature of each productacquired by the picture feature acquisition unit in association witheach product ID; a similarity calculation unit that calculates asimilarity between the products by using the picture feature of eachproduct stored in the feature database as a feature for similaritycalculation; a similarity product database that stores the similaritybetween the products calculated by the similarity calculation unit inassociation with each product ID; a unit that receives a similaritypicture searching request from a user terminal; a similarity picturesearching unit that extracts one or more products having a highsimilarity with the product from the similarity product database byusing the product ID of the product which is a source of the similaritypicture searching request, as a key; and a unit that allows the userterminal to display the product picture and the product informationcorresponding to the product extracted by the similarity picturesearching unit.

JP2012-014544A discloses a coordination recommendation apparatus thatrecommends another type of item suitable for a combination with an inputitem so as to coordinate a plurality of types of fashion items, theapparatus including: a storage unit that stores a reference photo setpredetermined as a coordination picture set as a model and arecommendation photo set predetermined as a recommended item pictureset; a reference whole-body photo feature extraction unit that extractsa whole body photo representing a picture obtained by combining aplurality of types of items from the reference photo set, specifies anarea of each item in the extracted whole-body photo, and extracts afeature of the picture from the specified item area, and stores thefeature extracted for each item as a reference whole-body photo featureset in the storage unit; a recommendation photo feature extraction unitthat extracts a whole-body photo representing a picture in which aplurality of types of items are combined with each other and a singleitem photo representing a picture including a single item from therecommendation photo set, specifies an area of each item in theextracted whole-body photo, extracts a feature of the picture from thespecified item area, extracts a feature of the picture from the singleitem photo, and stores, as a recommendation photo feature set, thefeatures extracted for each item from the whole-body photo and thesingle item photo in the storage unit; and a recommendation unit thatlearns a relationships between item areas using the reference whole-bodyphoto feature set and the recommendation photo feature set, searchesanother type of item suitable for a combination with the input item fromthe recommendation photo set according to the feature of the picture ofthe input item, and suggests the searched item as a recommendation item.

SUMMARY

Among users, there are some users who do not decide specific images suchas a desired shape and a desired color of a product but have only anambiguous request such as a desire to buy a product which may be blendedwith a landscape. In this case, in order to search a product which maybe blended with the landscape, it is necessary to estimate an image ofthe landscape.

Aspects of non-limiting embodiments of the present disclosure relate toproviding an information processing apparatus and a non-transitorycomputer readable medium storing an information processing programcapable of estimating an image represented by a captured landscape.

Aspects of certain non-limiting embodiments of the present disclosureaddress the above advantages and/or other advantages not describedabove. However, aspects of the non-limiting embodiments are not requiredto address the advantages described above, and aspects of thenon-limiting embodiments of the present disclosure may not addressadvantages described above.

According to an aspect of the present disclosure, there is provided aninformation processing apparatus including: a receiving unit thatreceives an input picture obtained by capturing a landscape including anobject; and an estimation unit that estimates an image represented bythe landscape appearing in the input picture based on a learning modelin which the input picture received by the receiving unit is input.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiment(s) of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a diagram illustrating a configuration example of aninformation processing system according to a first exemplary embodiment;

FIG. 2 is a diagram illustrating a functional configuration example ofthe information processing apparatus according to the first exemplaryembodiment;

FIG. 3 is a diagram illustrating a configuration example of a main partof an electric system in the information processing apparatus;

FIG. 4 is a flowchart illustrating an example of a flow of estimationprocessing according to the first exemplary embodiment;

FIG. 5 is a diagram illustrating an example of an input picture;

FIG. 6 is a diagram illustrating a configuration example of theinformation processing system according to a second exemplaryembodiment;

FIG. 7 is a diagram illustrating a functional configuration example ofthe information processing apparatus according to the second exemplaryembodiment;

FIG. 8 is a flowchart illustrating an example of a flow of estimationprocessing according to the second exemplary embodiment; and

FIG. 9 is a diagram illustrating an example of a screen displayed on auser terminal.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings. In allthe drawings, components and processing having the same functions aredenoted by the same reference numerals, and repeated explanations willbe omitted.

First Exemplary Embodiment

FIG. 1 is a diagram illustrating a configuration example of aninformation processing system 100 according to the present exemplaryembodiment. The information processing system 100 includes aninformation processing apparatus 10 and a user terminal 20, and theinformation processing apparatus 10 and the user terminal 20 areconnected to each other via a communication line 2.

The information processing apparatus 10 receives a picture obtained bycapturing a landscape from the user terminal 20 via the communicationline 2, estimates an image represented by the landscape appearing in thepicture, and outputs an estimation result to the user terminal 20 viathe communication line 2.

Here, “image” is a feeling that a user has. For example, an imagerepresented by a landscape is a feeling that a user has for thelandscape in a case where the user views the landscape. In addition, afeeling that a user has for an interior of a living space is called“interior image”.

Examples of the image include, for example, a Northern-European modernimage which is a simple and non-tiring image using a texture of a woodand emphasizing functionality, a Northern-European natural image whichis an image using a texture of a natural wood as it is, and aNorthern-European vintage image which is an image in which a vintageinterior is included in a simple Northern-European interior using atexture of a wood. Further, as another example, for example, there arealso a Shabby-chic image which is an image representing a feminineatmosphere using decorations such as a curved line and a flower pattern,an elegant image which is an image representing elegance and grace, anda simple basic image which is an image in which the number of livinggoods such as furniture is reduced and the number of colors anddecorations used in living goods are decreased.

The user terminal 20 is an information device that transmits a pictureobtained by capturing a landscape to the information processingapparatus 10 and displays an estimation result of an image representedby the landscape, the estimation result being output from theinformation processing apparatus 10. As described above, the imagerepresented by the landscape is formed by an atmosphere of an interiorof a living space. Thus, the picture transmitted from the user terminal20 to the information processing apparatus 10 is, for example,preferably a picture obtained by capturing a person's living space suchas an inside space of a building or an appearance of a building. Inother words, for example, in a case where the picture is a pictureobtained by capturing only a single object in an inside space of abuilding, it is difficult to estimate an image of the entire livingspace from the picture, and thus the picture transmitted from the userterminal 20 to the information processing apparatus 10 includes, forexample, preferably a plurality of objects forming an image of a livingspace, that is, a plurality of objects having a common image (morepreferably, different types of objects).

The user terminal 20 is not limited to a specific type of informationdevices as long as the user terminal 20 has a communication function oftransmitting and receiving information to and from the informationprocessing apparatus 10 via the communication line 2, and a notificationfunction of notifying an operator of the user terminal 20 (hereinafter,referred to as “user”) of information using at least one notificationform among a text, a picture, a sound, and a vibration. For example, asthe user terminal 20, in addition to a desktop computer, a portabletablet computer, a smartphone, and a wearable device to be worn on auser's body such as an arm or a face, a smart speaker (AI speaker) orthe like which performs conversation with a user by using artificialintelligence and also performs transmission and reception of picturesmay be used.

In the example of FIG. 1, although only one user terminal 20 is includedin the information processing system 100, the number of the userterminals 20 is not limited, and a plurality of user terminals 20 may beincluded in the information processing system 100.

The communication line 2 may be a wireless channel or a wired line, andmay be a dedicated line or a public line to which many unspecifieddevices are connected. Needless to say, in a case where the userterminal 20 is an information device such as a smartphone that moveswith a user, the user terminal 20 is, for example, preferably connectedto the information processing apparatus 10 via a wireless channel.

FIG. 2 is a diagram illustrating a functional configuration example ofthe information processing apparatus 10. The information processingapparatus 10 includes a receiving unit 12, an estimation unit 14, and anoutput unit 18.

The receiving unit 12 receives a picture obtained by capturing alandscape from the user terminal 20 via the communication line 2.Hereinafter, in some cases, the picture received from the user terminal20 maybe referred to as “input picture”. The receiving unit 12 transmitsthe received input picture to the estimation unit 14.

The estimation unit 14 includes an image estimation unit 14A, a placeestimation unit 14B, and a color estimation unit 14C, each of whichestimates a predetermined item from contents of the input picture.

In a case where the input picture is received, the image estimation unit14A estimates an image represented by the landscape appearing in theinput picture. For estimation of the image represented by the landscapeappearing in the input picture, for example, a machine learning methodis used. The machine learning method is an estimation method of findingregularity which potentially exists between data and meaningsrepresented by the data by repeatedly learning the data and the meaningsrepresented by the data using a plurality of pieces of data, the dataand the meanings being correlated with each other, and estimatingmeanings represented by unknown data based on the regularity.

More specifically, by inputting a plurality of sample pictures obtainedby capturing landscapes different from that of the input picture andsetting, as an output, an image represented by objects included in thelandscape of each sample picture, an image learning model in whichcorrelation between a landscape and an image is learned in advance isobtained. Thus, the image estimation unit 14A estimates an imagerepresented by the landscape appearing in the input picture by inputtingthe input picture into the image learning model obtained by learning.

The image learning model is obtained, for example, by using a known deeplearning technique which is an example of a machine learning method asdescribed in NPL 1 to NPL 3.

NPL 1: A. Krizhevsky, I. Sutskever, G. E. Hinton, “Imagenetclassification with deep convolutional neural networks” In Proc. ofNIPS, 2012.

NPL 2: K. Simonyan and A. Zisserman, “Very deep convolutional networksfor large-scale image recognition” In Proc. of ICLR, 2015.

NPL 3: K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition” In Proc. of CVPR, 2016.

In the image learning model, as an algorithm for correlating a samplepicture with an image represented by the sample picture, for example, aback propagation method as described in NPL 4 is used.

NPL 4: D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learningrepresentations by back-propagating errors” Nature 323 (6088) p533-p536,1986.

That is, by using the image learning model, the image estimation unit14A estimates an image represented by the landscape appearing in theinput picture based on features of the objects included in the inputpicture, such as shapes, arrangements, colors, and sizes of the objects,the features forming an atmosphere of the landscape represented in theinput picture. The image learning model in which correlation between alandscape and an image is learned in advance is an example of a learningmodel.

The image estimation result of the image estimation unit 14A isrepresented by an M-dimensional real number vector S as shown inEquation (1).

S=(s ₁ , s ₂ , . . . , s _(M))^(T)  (1)

Here, “M (M is an integer of one or more)” indicates the number ofimages that may be estimated by the image estimation unit 14A, that is,the number of learned images, and a real number s_(m) (m=1 to M)indicates a probability that the landscape appearing in the inputpicture represents an m-th image. Each different image that may beestimated by the image estimation unit 14A is correlated with a variablem, and as a value of s_(m) increases, the image represented by thevariable m is more emphasized in the landscape. Thus, for example, anoutput layer of the image learning model is formed of M units, and anoutput value of each unit is correlated with s_(m). “T” is a numericalsymbol representing a transposed matrix.

In a case where the input picture is received, the place estimation unit14B estimates a place of the landscape appearing in the input picture.Here, “place” is a name of a living space classified according to use,and is represented by being classified into, for example, a living room,a dining room, a bedroom, a veranda, and an entrance.

Similar to the image estimation unit 14A, the place estimation unit 14Buses, for example, a machine learning method for estimating the place ofthe landscape appearing in the input picture.

More specifically, by inputting the plurality of sample picturesobtained by capturing landscapes different from that of the inputpicture and setting, as an output, a place represented by the landscapeof each sample picture, a place learning model in which correlationbetween a landscape and a place is learned in advance is obtained. Thus,the place estimation unit 14B estimates a place of the landscapeappearing in the input picture by inputting the input picture into theplace learning model obtained by learning.

The place learning model is also obtained, for example, by using a knowndeep learning technique as described in NPL 1 to NPL 3. Similar to theimage learning model, even in the place learning model, as an algorithmfor correlating a sample picture with a place of a landscape representedby the sample picture, for example, a back propagation method asdescribed in NPL 4 is used. The place learning model in whichcorrelation between a landscape and a place is learned in advance is anexample of another learning model different from the image learningmodel.

The place estimation result of the place estimation unit 14B isrepresented by a K-dimensional real number vector R as shown in Equation(2).

R=(r ₁ , r ₂ , . . . , r _(K))^(T)  (2)

Here, “K (K is an integer of one or more)” indicates the number ofplaces that may be estimated by the place estimation unit 14B, that is,the number of learned places, and a real number r_(k) (k=1 to K)indicates a probability that the landscape appearing in the inputpicture represents a k-th place. Each different place that may beestimated by the place estimation unit 14B is correlated with a variablek, and as a value of r_(k) increases, the place correlated with thevariable k is more emphasized in the landscape. Thus, for example, anoutput layer of the place learning model is formed of K units, and anoutput value of each unit is correlated with r_(k).

In a case where the input picture is received, the color estimation unit14C estimates a difference color with respect to a color of thelandscape appearing in the input picture. “Difference color” refers to acolor on which a line of sight is focused in a case where the landscapeis colored, that is, an accent color. The difference color has an effectwhich makes a change in the landscape or enhances a color of thelandscape. For example, a complementary color with respect to the colorof the landscape is an example of the difference color.

The color estimation unit 14C represents the estimated difference color,as a numerical value. Any color system may be used as long as thedifference color is uniquely represented as a numerical value. On theother hand, as an example, it is assumed that the color estimation unit14C uses a Lab color system for expressing colors. A specific method ofestimating the difference color in the color estimation unit 14C will bedescribed later.

The estimation unit 14 transmits the estimation results on the image,the place, and the difference color, which are respectively estimated bythe image estimation unit 14A, the place estimation unit 14B, and thecolor estimation unit 14C, to the output unit 18.

In a case where the estimation results are received from the estimationunit 14, the output unit 18 outputs each of the estimation resultsestimated by the estimation unit 14 to the user terminal 20.

FIG. 3 is a diagram illustrating a configuration example of a main partof an electric system in the information processing apparatus 10. Theinformation processing apparatus 10 is configured with, for example, acomputer 40.

The computer 40 includes a central processing unit (CPU) 41 thatcorresponds to each unit of the information processing apparatus 10according to the present exemplary embodiment illustrated in FIG. 2, aread only memory (ROM) 42 that stores an information processing program,a random access memory (RAM) 43 that is used as a temporary work area ofthe CPU 41, a non-volatile memory 44, and an input and output interface(I/O) 45. The CPU 41, the ROM 42, the RAM 43, the non-volatile memory44, and the I/O 45 are connected to each other via a bus 46.

The non-volatile memory 44 is an example of a storage device in whichstored information is maintained even in a case where power supplied tothe non-volatile memory 44 is cut off, and as the non-volatile memory44, for example, a semiconductor memory is used. On the other hand, ahard disk may be used as the non-volatile memory 44. The non-volatilememory 44 is not necessarily provided in the computer 40, and a storagedevice configured to be detachably connected to the computer 40, such asa memory card, may be used.

On the other hand, for example, a communication unit 47, an input unit48, and a display unit 49 are connected to the I/O 45.

The communication unit 47 is connected to the communication line 2, andincludes a communication protocol for performing data communication withthe user terminal 20 connected to the communication line 2 and anexternal device not illustrated.

The input unit 48 is an input device that receives an instruction froman operator of the information processing apparatus 10 and transmits thereceived instruction to the CPU 41. As the input unit 48, for example, abutton, a keyboard, a mouse, a touch panel, and the like are used.

The display unit 49 is a display device that displays informationprocessed by the CPU 41 as a picture. As the display unit 49, forexample, a liquid crystal display, an organic electro luminescence (EL)display, or the like is used.

Units that may be connected to the I/O 45 are not limited to the unitsillustrated in FIG. 3. For example, a sound output unit that outputsinformation by a sound or a printing unit that prints information on arecording medium such as paper may be connected to the I/O 45.

Next, an operation of the information processing apparatus 10 will bedescribed with reference to FIG. 4.

FIG. 4 is a flowchart illustrating an example of a flow of estimationprocessing executed by the CPU 41 in a case where the informationprocessing apparatus 10 is activated.

The information processing program that prescribes estimation processingis stored in advance, for example, in the ROM 42 of the informationprocessing apparatus 10. The CPU 41 of the information processingapparatus 10 reads the information processing program stored in the ROM42, and executes estimation processing. It is assumed that each of theimage learning model and the place learning model obtained by learningis stored in advance in the non-volatile memory 44 of the informationprocessing apparatus 10.

First, in step S10, the CPU 41 determines whether or not an inputpicture is received from the user terminal 20. In a case where an inputpicture is not received, the CPU 41 monitors a reception state of aninput picture by repeatedly executing the determination processing ofstep S10. On the other hand, in a case where an input picture isreceived, the received input picture is stored in the RAM 43, and theprocess proceeds to step S20.

In step S20, the CPU 41 generates a real number vector S shown inEquation (1) by using output values of M units forming an output layerof the image learning model in a case where the input picture receivedin step S10 is input to the image learning model stored in thenon-volatile memory 44. The real number vector S is an estimation resultof an image represented by the landscape appearing in the input picture(also referred to as an “image of the landscape”), and the CPU 41 storesthe image estimation result in the RAM 43.

In step S30, the CPU 41 generates a real number vector R shown inEquation (2) by using output values of K units forming an output layerof the place learning model in a case where the input picture receivedin step S10 is input to the place learning model stored in thenon-volatile memory 44. The real number vector R is an estimation resultof a place of the landscape appearing in the input picture (in somecases, simply referred to as a “place of the landscape”) , and the CPU41 stores the place estimation result in the RAM 43.

In step S40, the CPU 41 estimates a difference color with respect to acolor of the landscape appearing in the input picture received in stepS10.

In general, an input picture includes a plurality of colors. As thenumber of colors increases, a time required for estimation of thedifference color becomes longer, and types of colors are more likely tobe dispersed. As a result, estimation accuracy of the difference colortends to be decreased. For this reason, the CPU 41 performs colorreduction processing of reducing the number of colors included in theinput picture to a predetermined number (for example, 256 colors) bytreating similar colors in which numerical values representing colorsare included within a predetermined range, as the identical color.

The CPU 41 specifies a color which is most frequently used in thecolor-reduced input picture, as a base color of the input picture. Afrequency of use of each color in the input picture is represented, forexample, by a size of a display area for each color, and the displayarea of each color is measured by counting the number of pixels havingpixel values corresponding to each color. That is, the base color of theinput picture is a color which is most frequently used in the inputpicture, and is a representative color of the input picture.

In order to set a color of the input picture as a base color of theinput picture, a display area of the color is, for example, preferablyequal to or larger than an area in which the color may be visuallydetermined to be used more frequently than other colors (“thresholdarea”) . Thus, the CPU 41 sets, for example, preferably a color having adisplay area equal to or larger than the threshold area, as the basecolor of the input picture.

In other words, even in a case where a color is most frequently used inthe input picture, in a case where a display area of the color issmaller than the threshold area, the CPU 41 does not specify, forexample, preferably, the color as a base color of the input picture.

The CPU 41 may perform a control such that a display area of a color ofa predetermined object (hereinafter, referred to as “specific object”)is not included in a final display area of the color. Here, “specificobject” is not an object which is disposed to form a unified image of aliving space, is an object which is disposed for a healing purpose or anobject which autonomously moves in a plurality of living spaces, anddoes not contribute to formation of an image of a living space.

Thus, for example, a plant, a person, a pet, and a cleaning robot areexamples of the specific object, and a display area of a color of thespecific object is not used for calculation of a base color of the inputpicture.

The object appearing in the input picture maybe detected by using anobject detection model obtained by machine learning using, for example,a known deep learning technique as described in NPL 5. The objectdetection model obtained by learning is stored, for example, in thenon-volatile memory 44.

NPL 5: S. Ren, K. He, R. Girshick, J. Sun, “Faster R-CNN: towardsreal-time object detection with region proposal networks”, In Proc. ofNIPS, 2015.

In a case where a base color of the input picture is specified, the CPU41 estimates a color having a largest color difference from the basecolor in the Lab color system, as a difference color with respect to thelandscape appearing in the input picture. The difference color is notnecessarily one color, and the CPU 41 may set, for example, L colors (Lis an integer of two or more) in order from the color with the largestcolor difference from the base color, as difference colors.

In step S50, the CPU 41 outputs the estimation result of each item tothe user terminal 20 from which the input picture is transmitted, bycontrolling the communication unit 47, the input picture being used forthe estimation of an image of the landscape, a place of the landscape,and a difference color with respect to the landscape. Thus, the user mayobtain an image represented by the landscape appearing in the inputpicture, a place of the landscape, and a difference color with respectto the landscape. The difference color is represented by a numericalvalue of the Lab color system. Thus, in a case of outputting thedifference color to the user terminal 20, the CPU 41 may convert thenumerical value of the Lab color system into a color name such as “red”or “blue” which may be easily recognized by the user, and output thecolor name.

The output destination of the estimation result is not limited to theuser terminal 20 from which the input picture is transmitted. Forexample, the estimation result may be output to a user terminal 20different from a transmission source of the input picture and thedisplay unit 49 of the information processing apparatus 10, or may beoutput to another external device which is connected to thecommunication line 2 and is different from the user terminal 20.

Thereby, estimation processing illustrated in FIG. 4 is ended.

FIG. 5 is a diagram illustrating an example of the input picturereceived by the information processing apparatus 10 from the userterminal 20. In a case where an input picture (for example, a picture ofa room based on a color of natural wood) as illustrated in FIG. 5 isreceived, the information processing apparatus 10 outputs information,for example, “image=Northern-European modern”, “place=living room”, and“difference color=blue-violet”, to the user terminal 20, as theestimation results.

In the estimation processing illustrated in FIG. 4, each item of theimage of the landscape, the place of the landscape, and the differencecolor are estimated. On the other hand, the information processingapparatus 10 does not need to estimate all the items, and may estimateat least one item among the items.

In addition, in the estimation processing illustrated in FIG. 4, in acase of estimating a place of the landscape, a name of a living space isoutput as the estimation result. On the other hand, in a case where theplace of the landscape is identified, the place of the landscape may berepresented by another classification.

For example, the information processing apparatus 10 may estimate anaddress of the landscape appearing in the input picture. In the case ofestimating an address, it is not necessary to estimate a street numberof a place represented by the landscape, and the address maybeestimated, for example, at a level of a city, a town, or a village. Inthis case, by inputting the plurality of sample pictures obtained bycapturing landscapes different from that of the input picture andsetting, as an output, an address of a place represented by thelandscape of each sample picture, the information processing apparatus10 may estimate a place using the place learning model in whichcorrelation between a landscape and an address is learned in advance.

As another example, in a case of receiving, as an input picture, apicture obtained by capturing the inside of a store, the informationprocessing apparatus 10 may estimate a store name. In this case, byinputting the plurality of sample pictures obtained by capturing theinsides of stores different from that of the input picture and setting,as an output, a store name represented by the landscape of each samplepicture, the information processing apparatus 10 may estimate a placeusing the place learning model in which correlation between a landscapeand a store name is learned in advance.

In addition, in the estimation processing illustrated in FIG. 4, thedifference color with respect to the landscape appearing in the inputpicture is estimated. On the other hand, a color estimated by theinformation processing apparatus 10 may be any color as long as thecolor is related to the input picture, and is not limited to thedifference color.

For example, the information processing apparatus 10 may estimate a basecolor of the input picture and a color of each object included in thelandscape of the input picture. Further, the information processingapparatus 10 may output the estimated color of each object in order ofdisplay areas of colors.

As described above, the information processing apparatus 10 according tothe present exemplary embodiment estimates at least one among an imageof the landscape, a place of the landscape, and a color related to thelandscape, from the input picture obtained by capturing the landscapeincluding objects, and outputs the estimation result.

Second Exemplary Embodiment

As described above, in the first exemplary embodiment, the informationprocessing apparatus 10, which estimates at least one among an image ofthe landscape, a place of the landscape, and a color related to thelandscape from the input picture, is described. On the other hand, theinformation processing apparatus 10 may suggest other informationobtained from the estimation result, to the user.

In a second exemplary embodiment, an information processing apparatus10A, which suggests a product matching with the landscape to the user byusing at least one item among the estimated image of the landscape, theestimated place of the landscape, and the estimated color related to thelandscape, will be described.

FIG. 6 is a diagram illustrating a configuration example of aninformation processing system 100A according to the present exemplaryembodiment. The information processing system 100A is different from theinformation processing system 100 illustrated in FIG. 1 in that theinformation processing apparatus 10 is replaced with an informationprocessing apparatus 10A and a product information database (DB) 30 isadded.

The information processing apparatus 10A receives an input picture fromthe user terminal 20 via the communication line 2, estimates at leastone item among an image represented by the landscape appearing in theinput picture, a place of the landscape, and a color related to thelandscape (for example, a difference color with respect to thelandscape), and outputs a product matching with the landscape to theuser terminal 20 via the communication line 2. In the followingdescription, an example in which the information processing apparatus10A estimates each item of an image of the landscape, a place of thelandscape, and a difference color with respect to the landscape, will bedescribed unless otherwise specified.

The product information DB 30 is a storage device which storesinformation of a product (hereinafter, referred to as “productinformation”) that the information processing apparatus 10A suggests, asa product matching with the landscape represented in the input picture,to the user. More specifically, a name of a product, a picture of aproduct, and a color of a product are stored in the product informationDB 30 in correlation with each product, as product information. On theother hand, the product information stored in the product information DB30 is not limited to the information, and any information related to aproduct, such as a price of a product, a supplier of a product, a sizeof a product, and the like, may be stored in the product information DB30 in correlation with each product. A color of a product is measuredusing, for example, a colorimeter, and is set as a color value of acolor system which is the same as the color system used in the colorestimation unit 14C. On the other hand, a color of a product may be setas a color name.

In addition, there is no restriction on a type of a product stored inthe product information DB 30, and any type of product information, forexample, household appliances, clothes, tableware, and the like may bestored. In the present exemplary embodiment, as an example, productinformation on interior such as a chair, a table, a sofa, a shelf, acloset, general goods, lighting, a curtain, a carpet, and the like isstored. In the following description, a picture obtained by capturing aproduct may be referred to as a “product picture” . A plurality ofproducts may be included in the product picture. On the other hand, inorder to manage colors of products in correlation with each product,colors of products included in the product picture are, for example,preferably common. In addition, it is assumed that the product pictureis captured at a place where use of products is expected. For example,in a case of a dining table, a picture obtained by capturing a diningspace as a background is stored as a product picture in the productinformation DB 30 in correlation with a dining table.

In a case where product information of a product stored in the productinformation DB 30 is selected by the information processing apparatus10A, the product information is transmitted to the informationprocessing apparatus 10A.

FIG. 7 is a diagram illustrating a functional configuration example ofthe information processing apparatus 10A. A functional configuration ofthe information processing apparatus 10A is different from thefunctional configuration example of the information processing apparatus10 illustrated in FIG. 2 in that the output unit 18 is replaced with anoutput unit 18A and a setting unit 16 is added, and other configurationsare the same as those in FIG. 2.

The setting unit 16 causes the estimation unit 14 to estimate an imagerepresented by a product included in the product picture and an expecteduse place of the product for each product, by inputting all the productpictures stored in the product information DB 30 into each of the imagelearning model and the place learning model obtained by learning.Hereinafter, an image represented by a product may be referred to as“image of a product”, and an expected use place of a product may bereferred to as “place of a product”.

In the case where the image learning model is used for estimating animage of a product, assuming that the number of products stored in theproduct information DB 30 is I (I is an integer of one or more) , animage of an i-th product (i=1 to I) (hereinafter, referred to as“product i”) is represented by an M-dimensional real number vector S_(i)as shown in Equation (3) .

S _(i)=(s _(i1) , s _(i2) , . . . , s _(iM))^(T)  (3)

Here, a real number s_(im), (m=1 to M) indicates a probability that aproduct i represents an m-th image. Each different image that may beestimated by the image estimation unit 14A is correlated with a variablem, and thus, as a value of s_(im), increases, the image represented bythe variable m is more emphasized in the product.

On the other hand, in the case where the place learning model is usedfor estimating a place of a product, a place of a product i isrepresented by a K-dimensional real number vector R_(i) as shown inEquation (4).

R _(i)=(r _(i1) , r _(i2) , . . . , r _(iK))^(T)  (4)

Here, a real number r_(ik) (k=1 to K) represents a probability that aplace of a product is a k-th place. As a value of r_(ik) increases, theproduct is expected to be used at a place correlated with a variable k.

The setting unit 16 sets a matching degree of a product with thelandscape in a case where the product represented in the product pictureis disposed in the landscape having the image estimated from the inputpicture, that is, a matching degree of a product with the landscaperepresented in the input picture, for each product.

The setting unit 16 represents a matching degree of a product with thelandscape represented in the input picture, as a numerical value called“score”. As the score increases, the matching degree of a product withthe landscape represented in the input picture becomes higher.

The score includes a score for each item estimated by the estimationunit 14. Specifically, the score includes an image score indicatingsimilarity between an image represented by the landscape represented inthe input picture and an image of a product, a place score indicatingsimilarity between a place of the landscape appearing in the inputpicture and a place of a product, and a color score indicatingsimilarity between a difference color with respect to the landscapeappearing in the input picture and a color of a product.

As similarity between an image represented by the landscape representedin the input picture and an image of a product increases, an atmosphereof the landscape and an atmosphere of the product become similar to eachother. Thus, in a case where the product is disposed in the landscape,the product is likely to be blended with the landscape withoutincompatibility. Therefore, by setting the image score to be higher asthe similarity increases, the setting unit 16 represents the matchingdegree of the product with the landscape represented in the inputpicture. In other words, as the image score, an indicator whichindicates a higher value as the matching degree of the product with thelandscape represented in the input picture increases, may be used.

As an example, an image score U_(i) ^(S) of a product i is defined usingcosine similarity as shown in Equation (5).

$\begin{matrix}{U_{i}^{S} = \frac{S \cdot S_{i}}{{S}{S_{i}}}} & (5)\end{matrix}$

Here, as shown in Equation (1), the real number vector S is an estimatedvalue of an image represented by the landscape, and as shown in theEquation (3), the real number vector S_(i) is an estimated value of animage of a product i. “·” indicates inner product of vectors, and “| |”indicates a vector length.

Further, as the similarity between a place of the landscape appearing inthe input picture and a place of a product increases, a place at whichthe product is disposed becomes similar to a use place of the productthat is expected in advance. Thus, in a case where the product isdisposed in the landscape, the product is likely to be blended with thelandscape without incompatibility. Therefore, by setting the place scoreto be higher as the similarity increases, the setting unit 16 representsthe matching degree of the product with the landscape represented in theinput picture. In other words, as the place score, an indicator whichindicates a higher value as the matching degree of the product with thelandscape represented in the input picture increases, may be used.

Therefore, similar to the image score U_(i) ^(S), a place score U_(i)^(R) of a product i is defined using cosine similarity as shown inEquation (6).

$\begin{matrix}{U_{i}^{R} = \frac{R \cdot R_{i}}{{R}{R_{i}}}} & (6)\end{matrix}$

Here, as shown in Equation (2), the real number vector R is an estimatedvalue of a place of the landscape, and as shown in the Equation (4), thereal number vector R_(i) is an estimated value of a place of a producti.

In addition, as the similarity between the difference color with respectto the landscape appearing in the input picture and a color of a productincreases, in a case where the product is disposed in the landscape, theproduct functions as an accent color in the landscape. Thus, the productis likely to enhance a color of the landscape. Therefore, by setting thecolor score to be higher as the similarity increases, the setting unit16 represents the matching degree of the product with the landscaperepresented in the input picture. In other words, as the color score, anindicator which indicates a higher value as the matching degree of theproduct with the landscape represented in the input picture increases,may be used.

As an example, a color score U_(i) ^(C) of a product i is defined usingan arithmetic expression as shown in Equation (7).

$\begin{matrix}{U_{i}^{C} = \left\{ \begin{matrix}1 & \left( {c_{i} = c} \right) \\0 & \left( {c_{i} \neq c} \right)\end{matrix} \right.} & (7)\end{matrix}$

Here, “c” indicates a difference color with respect to the landscapeappearing in the input picture, and “c_(i)” indicates a color of aproduct i.

The setting unit 16 sets a product score U_(i), which is a final scoreof a product i, for each product using an arithmetic expression as shownin Equation (8) including the image score U_(i) ^(S), the place scoreU_(i) ^(R), and the color score U_(i) ^(C).

U _(i) =αU _(i) ^(S) +βU _(i) ^(R) +γU _(i) ^(C)  (8)

Here, α, β, and γ indicate weight parameters of the image score U_(i)^(S), the place score U_(i) ^(R), and the color score U_(i) ^(C), andadjust a degree of influence of the image score U_(i) ^(S), the placescore U_(i) ^(R), and the color score U_(i) ^(C) on the product score U.

The arithmetic expression of the product score U_(i) is not limited toEquation (8), and may be an arithmetic expression defined such that theproduct score U_(i) also becomes higher as the total value of the scoresof each item estimated by the estimation unit 14 (in this case, theimage score U_(i) ^(S), the place score U_(i) ^(R), and the color scoreU_(i) ^(C)) increases.

The output unit 18A outputs a product matching with the landscapeappearing in the input picture, as a recommended product by referring tothe product score U_(i) of each product received from the setting unit16, and thus the recommended product is suggested to the user.

A configuration of a main part of an electric system in the informationprocessing apparatus 10A is the same as the configuration example of themain part of the electric system in the information processing apparatus10 illustrated in FIG. 3.

Next, an operation of the information processing apparatus 10A will bedescribed with reference to FIG. 8.

FIG. 8 is a flowchart illustrating an example of a flow of estimationprocessing executed by the CPU 41 in a case where the informationprocessing apparatus 10A is activated.

The flowchart of the estimation processing illustrated in FIG. 8 isdifferent from the flowchart of the estimation processing according tothe first exemplary embodiment illustrated in FIG. 4 in that processingof step S60 to step S90 is added instead of deleting processing of stepS50.

After the difference color with respect to the landscape appearing inthe input picture is estimated in step S40, processing of step S60 isexecuted.

In step S60, the CPU 41 selects one product among the products stored inthe product information DB 30, and acquires product information of theselected product. For convenience of explanation, it is assumed that theselected product is a product i.

Here, an example in which the product information is stored in theproduct information DB 30 will be described. On the other hand, theproduct information may be stored in the non-volatile memory 44 of theinformation processing apparatus 10A. In this case, the productinformation DB 30 is not necessary in the information processing system100A.

In step S70, the CPU 41 extracts a product picture and a color of theproduct from the product information acquired in step S60, andcalculates a score for each item estimated in step S20 to step S40.

Specifically, the CPU 41 acquires a real number vector S_(i)representing an image of the product i by inputting the product pictureof the product i extracted from the product information into the imagelearning model obtained by learning. The CPU 41 calculates an imagescore U_(i) ^(S) of the selected product i according to Equation (5) byusing a real number vector S representing an image of the landscapeestimated in step S20 and a real number vector S_(i) representing animage of the product i. The real number vector S_(i) of the product imay be set in advance by a person based on an appearance of the productwithout using the image learning model. In this case, the real numbervector S_(i) which is set in advance may be included in the productinformation.

In addition, the CPU 41 acquires a real number vector R_(i) representinga place of the product i by inputting the product picture of the producti extracted from the product information into the place learning modelobtained by learning. The CPU 41 calculates a place score U_(i) ^(R) ofthe selected product according to Equation (6) by using a real numbervector R representing a place of the landscape estimated in step S30 anda real number vector R_(i) representing a place of the product i. Thereal number vector R_(i) of the product i may be set in advance by aperson based on an appearance of the product without using the placelearning model. In this case, the real number vector R_(i) which is setin advance may be included in the product information.

In addition, the CPU 41 calculates a color score U_(i) ^(C) of theselected product i according to Equation (7) by using a difference colorc with respect to the landscape estimated in step S40 and a color of theproduct i extracted from the product information.

In a case where a plurality of difference colors c with respect to thelandscape are estimated in the estimation processing of step S40, in acase where the color of the product i is identical to any one of thedifference colors c, the color score U_(i) ^(C) may be set to “1”.Further, in a case where the base color of the input picture isestimated in the estimation processing of step S40, in a case where thecolor of the product i is identical to the base color, the color scoreU_(i) ^(C) may be set to “1”.

Further, the CPU 41 calculates a product score U_(i) of the selectedproduct i according to Equation (8) by using the calculated image scoreU_(i) ^(S), the calculated place score U_(i) ^(R), and the calculatedcolor score U_(i) ^(C).

The weight parameters α, β, and γ used for calculation of the productscore U_(i) are stored in advance, for example, in the non-volatilememory 44. The weight parameters α, β, and γ are adjusted in advancesuch that the product score U_(i) of each product on a plurality ofsample pictures obtained by capturing landscapes different from that ofthe input picture becomes similar to the matching degree of the productwith the landscape of each of the sample pictures determined by afeeling of a person.

In step S80, the CPU 41 determines whether or not product information ofall the products stored in the product information DB 30 is acquired.

In a case where there is product information which is not acquired, theprocess proceeds to step S60. In this case, the CPU 41 selects oneproduct of which the product information is not yet acquired among theproducts stored in the product information DB 30, and acquires productinformation of the selected product. That is, by repeatedly executingthe processing of step S60 to step S80 until all the products stored inthe product information DB 30 are selected, the product score U_(i) ofeach product i stored in the product information DB 30 is calculated.

On the other hand, in a case where the product information of all theproducts stored in the product information DB 30 is acquired, theprocess proceeds to step S90.

In step S90, the CPU 41 outputs, to the user terminal 20 as atransmission source of the input picture, a screen on which recommendedproducts matching with the landscape of the input picture are displayedin order from the product with the highest product score U_(i) byreferring to the product score U_(i) of each product i stored in theproduct information DB 30, the product score being calculated in stepS70.

FIG. 9 is a diagram illustrating an example of a screen displayed on theuser terminal 20. As illustrated in FIG. 9, recommended productsmatching with the landscape appearing in the picture are displayed onthe user terminal 20, the picture being transmitted to the informationprocessing apparatus 10A by the user.

The recommended product is a product matching with the landscapeappearing in the input picture, and thus, various types of products suchas a chair and a table are displayed on the user terminal 20, asrecommended products. Therefore, the CPU 41 may incorporate a narrowingsearch function of narrowing the types of the recommended products intothe screen. For example, in a case where a chair type is specified bynarrowing search, the CPU 41 outputs, to the user terminal 20 as atransmission source of the input picture, a screen which displays onlychairs in order from the chair with the highest product score U_(i) byreferring to the product score U_(i) of the product classified as achair type.

The output destination of the recommended products is not limited to theuser terminal 20 from which the input picture is transmitted. Forexample, the recommended products may be output to a user terminal 20different from a transmission source of the input picture and thedisplay unit 49 of the information processing apparatus 10A, or may beoutput to another external device which is connected to thecommunication line 2 and is different from the user terminal 20.

In addition, there is also no restriction on an output form of therecommended product. For example, the recommended product may be outputto the user in a form using only a text such as a product name, or maybe output to the user in a form including a picture such as a productpicture in addition to a text. In some cases, the recommended productmay be output to the user by using a sound.

Thereby, estimation processing illustrated in FIG. 8 is ended.

In the estimation processing illustrated in FIG. 8, each item of theimage of the landscape, the place of the landscape, and the differencecolor with respect to the landscape is estimated, and the recommendedproduct is output based on the matching degree of the product and eachestimation item. On the other hand, as described above, the estimationitems used for calculating the matching degree of the product are notlimited to the image of the landscape, the place of the landscape, andthe difference color with respect to the landscape. The matching degreeof the product with the landscape appearing in the input picture may becalculated using at least one of the estimation items, and therecommended product may be output.

For example, in a case where the image of the landscape and thedifference color with respect to the landscape are used, the productscore U_(i) of the product i is represented by Equation (9) , and in acase where the place of the landscape and the difference color withrespect to the landscape are used, the product score U_(i) of theproduct i is represented by Equation (10).

U _(i) =αU _(i) ^(S) +γU _(i) ^(C)  (9)

U _(i) =βU _(i) ^(R) +γU _(i) ^(C)  (10)

Further, in the estimation processing illustrated in FIG. 8, in a caseof calculating the product score U_(i), the same weight parameters α, β,and γ are used for all the products. On the other hand, a combination ofthe weight parameters α, β, and γ with values different from each othermay be prepared for each type of the products, and the values of theweight parameters α, β, and γ used for calculation of the product scoreU_(i) may be changed depending on the type of the product. Thereby, theinformation processing apparatus 10A may adjust a degree of influence ofthe matching degree of the product with each estimation item on settingof the product score U_(i).

For example, general goods tend to be used for giving an accent color tothe landscape. Thus, in a case where the type of the product is generalgoods, the information processing apparatus 10A sets the weightparameter γ such that the matching degree of the color of the producthas a greater influence than that in other types of products on settingof the product score U_(i). Therefore, a product better matching withthe landscape appearing in the input picture is output as a recommendedproduct.

Although the present invention has been described with reference to eachexemplary embodiment, the present invention is not limited to the scopedescribed in each exemplary embodiment. Various modifications orimprovements may be added to each exemplary embodiment without departingfrom the spirit of the present invention. Also, an exemplary embodimentobtained by adding the modifications or improvements falls within atechnical scope of the present invention. For example, the order ofprocessing may be changed without departing from the spirit of thepresent invention.

In each exemplary embodiment, as an example, the estimation processingis realized by software. On the other hand, for example, in a case wherean application specific integrated circuit (ASIC) is provided, the sameprocessing as the flowcharts illustrated in FIG. 4 and FIG. 8 may beprocessed by hardware. In this case, as compared to the case where theestimation processing is realized by software, it is possible to performthe processing at high speed.

Further, in each exemplary embodiment, although a form in which theinformation processing program is installed in the ROM 42 has beendescribed, the present invention is not limited thereto. The informationprocessing program according to the present invention may also beprovided by being recorded in a computer-readable storage medium. Forexample, the information processing program according to the presentinvention may be provided by being recorded on an optical disc such as acompact disc (CD)-ROM or digital versatile disc (DVD)-ROM. Further, theinformation processing program according to the present invention may beprovided by being recorded in a semiconductor memory such as a UniversalSerial Bus (USB) memory and a flash memory. Furthermore, the informationprocessing program according to the present invention may be acquiredfrom an external device (not illustrated) connected to the communicationline 2 via the communication line 2.

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing apparatus comprising: a receiving unit that receives an input picture obtained by capturing a landscape including an object; and an estimation unit that estimates an image represented by the landscape appearing in the input picture based on a learning model in which the input picture received by the receiving unit is input.
 2. The information processing apparatus according to claim 1, wherein the receiving unit receives, as the input picture, a picture obtained by capturing a landscape on an appearance of a building or inside a building.
 3. The information processing apparatus according to claim 2, wherein the receiving unit receives, as the input picture, a picture obtained by capturing a landscape on an appearance of a building or inside a building, the landscape including a plurality of objects having a common image.
 4. The information processing apparatus according to claim 1, wherein the estimation unit further estimates a place of the landscape appearing in the input picture by inputting the input picture received by the receiving unit into another learning model different from the learning model.
 5. The information processing apparatus according to claim 2, wherein the estimation unit further estimates a place of the landscape appearing in the input picture by inputting the input picture received by the receiving unit into another learning model different from the learning model.
 6. The information processing apparatus according to claim 3, wherein the estimation unit further estimates a place of the landscape appearing in the input picture by inputting the input picture received by the receiving unit into another learning model different from the learning model.
 7. The information processing apparatus according to claim 1, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 8. The information processing apparatus according to claim 2, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 9. The information processing apparatus according to claim 3, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 10. The information processing apparatus according to claim 4, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 11. The information processing apparatus according to claim 5, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 12. The information processing apparatus according to claim 6, wherein the estimation unit further estimates a difference color with respect to the landscape appearing in the input picture.
 13. The information processing apparatus according to claim 7, wherein the estimation unit does not use a color of a predetermined object for estimating the difference color.
 14. The information processing apparatus according to claim 13, wherein the predetermined object is set as an animal or a plant.
 15. The information processing apparatus according to claim 1, further comprising: a setting unit that sets, by using a plurality of product pictures obtained by capturing products and prepared in advance, a matching degree of each product included in the product picture with the image estimated by the estimation unit; and an output unit that outputs, as a recommended product, the product included in the product picture in order from the product having the highest matching degree which is set by the setting unit.
 16. The information processing apparatus according to claim 15, wherein, in a case where a place at which the landscape appearing in the input picture is captured is estimated by the estimation unit, the setting unit sets the matching degree of each product such that the matching degree increases as the product is made in expectation of use at the place estimated by the estimation unit.
 17. The information processing apparatus according to claim 15, wherein, in a case where a difference color with respect to the landscape appearing in the input picture is estimated by the estimation unit, the setting unit sets the matching degree of each product such that the matching degree increases as the product has a color closer to the difference color estimated by the estimation unit.
 18. The information processing apparatus according to claim 16, wherein the setting unit adjusts a degree of influence of a plurality of items on setting of the matching degree according to a type of the product included in the product picture, the matching degree being set using the plurality of items including the image estimated by the estimation unit.
 19. The information processing apparatus according to claim 17, wherein, in a case where a type of the product included in the product picture is general goods, the setting unit sets a degree of influence of a color of the product on setting of the matching degree to be higher than degrees of influence of colors of other types of products on setting of the matching degree.
 20. A non-transitory computer readable medium storing an information processing program causing a computer to function as each unit of the information processing apparatus according to claim
 1. 